Getting the most out of R

# Getting the most out of R
## Physalia Courses
### Luis D. Verde Arregoitia
### February 2021

---

.dk-section-title h1 {
  color: #f3f3f3;
  font-size: 100px;
  line-height: 1;
}

.smallcode .remark-code { /*Change made here*/
  font-size: 75% !important;
}

.tinycode .remark-code { /*Change made here*/
  font-size: 50% !important;
}

</style>

## Wednesday - Increasing efficiency

- Iteration, writing loops and using {`purrr`}

- Apply functions to many things at once

- Reading many files at once

- Modifying and exporting multiple objects

- ~~Useful addins and helpers~~

- Regular expressions for working with text strings

---

# Challenge

.f3.white[
" `Airports.csv`" contains yearly rankings for total number of passengers at the world's busiest airports across several countries between 2016-2020.

1. Find the busiest (highest ranked) airports for each country and year .f4[(some countries have multiple entries per year)]

2. Which countries had >1 different busiest airports (by airport code) in the time period covered by the data? 
]

---

```r
china <-  airports[airports$Country=="China",]
china_years <- china$Year

china2016 <- china[china$Year==2016,] 
china2016busiest <- 
  china2016[which(china2016$Rank==min(china2016$Rank)),]

china2017 <- china[china$Year==2017,] 
china2017busiest <- 
  china2017[which(china2017$Rank==min(china2017$Rank)),]

china2018 <- china[china$Year==2018,] 
china2018busiest <- 
  china2018[which(china2018$Rank==min(china2018$Rank)),]
```

---

## Iteration - perform a process repeatedly

Any time we copy and paste code, and then edit, we are adding unnecessary steps that add room for errors

.tr[
Nick Holt (2017)  
[Function Iterations in R](https://nick-holt.github.io/2017/12/function-iterations-in-r/)
]

---

## Grouped data

.f3[Most data analysis and manipulations tasks can be approached using the .b.purple[“split-apply-combine”] paradigm:]

.tr[
Data Carpentry (2018)  
[Aggregating and analyzing data with dplyr](https://datacarpentry.org/R-genomics/04-dplyr.html)

Hadley Wickham (2011)  
[The split-apply-combine strategy for data analysis](https://doi.org/10.18637/jss.v040.i01) 
]

---

.f2.code[.b.blue[`dplyr::`].red.b[`group_by()`]]

.f4[
- Define groups with one or more variables  
- Stratify a data frame 
- equivalent to `aggregate()` and the '`by = `' argument in .b[`data.table`]
]

.center[
<img align="middle" src="data:image/png;base64,#imgs/groupby.PNG" width="67%" />    
]
from: Manipulación de datos en R con dplyr - Rubén Sánchez"

---

## Aggregating or manipulating grouped data

.f3[
Once `group_by()` creates a "grouped" version of a table, `dplyr` functions will manipulate each "group" separately and then combine the results. 
]

.bottom[
.tr[
RStudio (2021)  
[Data transformation with dplyr cheatsheet](
https://raw.githubusercontent.com/rstudio/cheatsheets/main/data-transformation.pdf)
]
]

---

### Within groups, we can:

- extract the first or last _n_ rows from each group

- find the minimum or maximum values

- remove duplicated rows

- count records, etc.]

---

```r
head(msleep)
```

<div id="tufsfrfjrn" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;">
<style>html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#tufsfrfjrn .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#tufsfrfjrn .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#tufsfrfjrn .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#tufsfrfjrn .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 6px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#tufsfrfjrn .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#tufsfrfjrn .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#tufsfrfjrn .gt_col_heading {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#tufsfrfjrn .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#tufsfrfjrn .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#tufsfrfjrn .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#tufsfrfjrn .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 5px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#tufsfrfjrn .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#tufsfrfjrn .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#tufsfrfjrn .gt_from_md > :first-child {
  margin-top: 0;
}

#tufsfrfjrn .gt_from_md > :last-child {
  margin-bottom: 0;
}

#tufsfrfjrn .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#tufsfrfjrn .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#tufsfrfjrn .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#tufsfrfjrn .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#tufsfrfjrn .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#tufsfrfjrn .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#tufsfrfjrn .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#tufsfrfjrn .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#tufsfrfjrn .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#tufsfrfjrn .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#tufsfrfjrn .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#tufsfrfjrn .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#tufsfrfjrn .gt_left {
  text-align: left;
}

#tufsfrfjrn .gt_center {
  text-align: center;
}

#tufsfrfjrn .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#tufsfrfjrn .gt_font_normal {
  font-weight: normal;
}

#tufsfrfjrn .gt_font_bold {
  font-weight: bold;
}

#tufsfrfjrn .gt_font_italic {
  font-style: italic;
}

#tufsfrfjrn .gt_super {
  font-size: 65%;
}

#tufsfrfjrn .gt_footnote_marks {
  font-style: italic;
  font-weight: normal;
  font-size: 65%;
}
</style>
<table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 21px;">name</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 21px;">genus</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 21px;">vore</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 21px;">order</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" style="font-size: 21px;">conservation</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 21px;">sleep_total</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 21px;">sleep_rem</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 21px;">sleep_cycle</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 21px;">awake</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 21px;">brainwt</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" style="font-size: 21px;">bodywt</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td class="gt_row gt_left" style="font-size: 21px;">Cheetah</td>
<td class="gt_row gt_left" style="font-size: 21px;">Acinonyx</td>
<td class="gt_row gt_left" style="font-size: 21px;">carni</td>
<td class="gt_row gt_left" style="font-size: 21px;">Carnivora</td>
<td class="gt_row gt_left" style="font-size: 21px;">lc</td>
<td class="gt_row gt_right" style="font-size: 21px;">12.1</td>
<td class="gt_row gt_right" style="font-size: 21px;">NA</td>
<td class="gt_row gt_right" style="font-size: 21px;">NA</td>
<td class="gt_row gt_right" style="font-size: 21px;">11.9</td>
<td class="gt_row gt_right" style="font-size: 21px;">NA</td>
<td class="gt_row gt_right" style="font-size: 21px;">50.000</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 21px;">Owl monkey</td>
<td class="gt_row gt_left" style="font-size: 21px;">Aotus</td>
<td class="gt_row gt_left" style="font-size: 21px;">omni</td>
<td class="gt_row gt_left" style="font-size: 21px;">Primates</td>
<td class="gt_row gt_left" style="font-size: 21px;">NA</td>
<td class="gt_row gt_right" style="font-size: 21px;">17.0</td>
<td class="gt_row gt_right" style="font-size: 21px;">1.8</td>
<td class="gt_row gt_right" style="font-size: 21px;">NA</td>
<td class="gt_row gt_right" style="font-size: 21px;">7.0</td>
<td class="gt_row gt_right" style="font-size: 21px;">0.01550</td>
<td class="gt_row gt_right" style="font-size: 21px;">0.480</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 21px;">Mountain beaver</td>
<td class="gt_row gt_left" style="font-size: 21px;">Aplodontia</td>
<td class="gt_row gt_left" style="font-size: 21px;">herbi</td>
<td class="gt_row gt_left" style="font-size: 21px;">Rodentia</td>
<td class="gt_row gt_left" style="font-size: 21px;">nt</td>
<td class="gt_row gt_right" style="font-size: 21px;">14.4</td>
<td class="gt_row gt_right" style="font-size: 21px;">2.4</td>
<td class="gt_row gt_right" style="font-size: 21px;">NA</td>
<td class="gt_row gt_right" style="font-size: 21px;">9.6</td>
<td class="gt_row gt_right" style="font-size: 21px;">NA</td>
<td class="gt_row gt_right" style="font-size: 21px;">1.350</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 21px;">Greater short-tailed shrew</td>
<td class="gt_row gt_left" style="font-size: 21px;">Blarina</td>
<td class="gt_row gt_left" style="font-size: 21px;">omni</td>
<td class="gt_row gt_left" style="font-size: 21px;">Soricomorpha</td>
<td class="gt_row gt_left" style="font-size: 21px;">lc</td>
<td class="gt_row gt_right" style="font-size: 21px;">14.9</td>
<td class="gt_row gt_right" style="font-size: 21px;">2.3</td>
<td class="gt_row gt_right" style="font-size: 21px;">0.1333333</td>
<td class="gt_row gt_right" style="font-size: 21px;">9.1</td>
<td class="gt_row gt_right" style="font-size: 21px;">0.00029</td>
<td class="gt_row gt_right" style="font-size: 21px;">0.019</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 21px;">Cow</td>
<td class="gt_row gt_left" style="font-size: 21px;">Bos</td>
<td class="gt_row gt_left" style="font-size: 21px;">herbi</td>
<td class="gt_row gt_left" style="font-size: 21px;">Artiodactyla</td>
<td class="gt_row gt_left" style="font-size: 21px;">domesticated</td>
<td class="gt_row gt_right" style="font-size: 21px;">4.0</td>
<td class="gt_row gt_right" style="font-size: 21px;">0.7</td>
<td class="gt_row gt_right" style="font-size: 21px;">0.6666667</td>
<td class="gt_row gt_right" style="font-size: 21px;">20.0</td>
<td class="gt_row gt_right" style="font-size: 21px;">0.42300</td>
<td class="gt_row gt_right" style="font-size: 21px;">600.000</td></tr>
    <tr><td class="gt_row gt_left" style="font-size: 21px;">Three-toed sloth</td>
<td class="gt_row gt_left" style="font-size: 21px;">Bradypus</td>
<td class="gt_row gt_left" style="font-size: 21px;">herbi</td>
<td class="gt_row gt_left" style="font-size: 21px;">Pilosa</td>
<td class="gt_row gt_left" style="font-size: 21px;">NA</td>
<td class="gt_row gt_right" style="font-size: 21px;">14.4</td>
<td class="gt_row gt_right" style="font-size: 21px;">2.2</td>
<td class="gt_row gt_right" style="font-size: 21px;">0.7666667</td>
<td class="gt_row gt_right" style="font-size: 21px;">9.6</td>
<td class="gt_row gt_right" style="font-size: 21px;">NA</td>
<td class="gt_row gt_right" style="font-size: 21px;">3.850</td></tr>
  </tbody>
  
  
</table>
</div>

---

### maximum `brainwt` for each order

```r
msleep %>% group_by(order) %>% slice_max(brainwt,n=1)
```

```
## # A tibble: 17 × 11
## # Groups:   order [17]
##    name   genus vore  order conservation sleep_total sleep_rem sleep_cycle awake
##    <chr>  <chr> <chr> <chr> <chr>              <dbl>     <dbl>       <dbl> <dbl>
##  1 Tenrec Tenr… omni  Afro… <NA>                15.6       2.3      NA       8.4
##  2 Cow    Bos   herbi Arti… domesticated         4         0.7       0.667  20  
##  3 Gray … Hali… carni Carn… lc                   6.2       1.5      NA      17.8
##  4 Big b… Epte… inse… Chir… lc                  19.7       3.9       0.117   4.3
##  5 Giant… Prio… inse… Cing… en                  18.1       6.1      NA       5.9
##  6 North… Dide… omni  Dide… lc                  18         4.9       0.333   6  
##  7 Phala… Phal… <NA>  Dipr… <NA>                13.7       1.8      NA      10.3
##  8 Europ… Erin… omni  Erin… lc                  10.1       3.5       0.283  13.9
##  9 Rock … Proc… <NA>  Hyra… lc                   5.4       0.5      NA      18.6
## 10 Rabbit Oryc… herbi Lago… domesticated         8.4       0.9       0.417  15.6
## 11 Short… Tach… inse… Mono… <NA>                 8.6      NA        NA      15.4
## 12 Horse  Equus herbi Peri… domesticated         2.9       0.6       1      21.1
## 13 Human  Homo  omni  Prim… <NA>                 8         1.9       1.5    16  
## 14 Afric… Loxo… herbi Prob… vu                   3.3      NA        NA      20.7
## 15 Afric… Cric… omni  Rode… <NA>                 8.3       2        NA      15.7
## 16 Tree … Tupa… omni  Scan… <NA>                 8.9       2.6       0.233  15.1
## 17 Easte… Scal… inse… Sori… lc                   8.4       2.1       0.167  15.6
## # … with 2 more variables: brainwt <dbl>, bodywt <dbl>
```
]

---

### one row sampled randomly from each order

```r
msleep %>% group_by(order) %>% slice_sample(n=1)
```

```
## # A tibble: 19 × 11
## # Groups:   order [19]
##    name   genus vore  order conservation sleep_total sleep_rem sleep_cycle awake
##    <chr>  <chr> <chr> <chr> <chr>              <dbl>     <dbl>       <dbl> <dbl>
##  1 Tenrec Tenr… omni  Afro… <NA>                15.6       2.3      NA       8.4
##  2 Sheep  Ovis  herbi Arti… domesticated         3.8       0.6      NA      20.2
##  3 Red f… Vulp… carni Carn… <NA>                 9.8       2.4       0.35   14.2
##  4 Bottl… Turs… carni Ceta… <NA>                 5.2      NA        NA      18.8
##  5 Littl… Myot… inse… Chir… <NA>                19.9       2         0.2     4.1
##  6 Long-… Dasy… carni Cing… lc                  17.4       3.1       0.383   6.6
##  7 Thick… Lutr… carni Dide… lc                  19.4       6.6      NA       4.6
##  8 Phala… Phal… <NA>  Dipr… <NA>                13.7       1.8      NA      10.3
##  9 Deser… Para… <NA>  Erin… lc                  10.3       2.7      NA      13.7
## 10 Tree … Dend… herbi Hyra… lc                   5.3       0.5      NA      18.7
## 11 Rabbit Oryc… herbi Lago… domesticated         8.4       0.9       0.417  15.6
## 12 Short… Tach… inse… Mono… <NA>                 8.6      NA        NA      15.4
## 13 Brazi… Tapi… herbi Peri… vu                   4.4       1         0.9    19.6
## 14 Three… Brad… herbi Pilo… <NA>                14.4       2.2       0.767   9.6
## 15 Chimp… Pan   omni  Prim… <NA>                 9.7       1.4       1.42   14.3
## 16 Asian… Elep… herbi Prob… en                   3.9      NA        NA      20.1
## 17 Mole … Spal… <NA>  Rode… <NA>                10.6       2.4      NA      13.4
## 18 Tree … Tupa… omni  Scan… <NA>                 8.9       2.6       0.233  15.1
## 19 Lesse… Cryp… omni  Sori… lc                   9.1       1.4       0.15   14.9
## # … with 2 more variables: brainwt <dbl>, bodywt <dbl>
```
]

---
.f3.b.blue[summarize]

```r
msleep %>% group_by(order) %>% 
  summarize(mean_slp = mean(sleep_total),
            mean_brainwt=mean(brainwt))
```
.center[
.tinycode[

```
## # A tibble: 19 × 3
##    order           mean_slp mean_brainwt
##    <chr>              <dbl>        <dbl>
##  1 Afrosoricida       15.6      0.0026  
##  2 Artiodactyla        4.52    NA       
##  3 Carnivora          10.1     NA       
##  4 Cetacea             4.5     NA       
##  5 Chiroptera         19.8      0.000275
##  6 Cingulata          17.8      0.0459  
##  7 Didelphimorphia    18.7     NA       
##  8 Diprotodontia      12.4     NA       
##  9 Erinaceomorpha     10.2      0.00295 
## 10 Hyracoidea          5.67     0.0152  
## 11 Lagomorpha          8.4      0.0121  
## 12 Monotremata         8.6      0.025   
## 13 Perissodactyla      3.47     0.414   
## 14 Pilosa             14.4     NA       
## 15 Primates           10.5     NA       
## 16 Proboscidea         3.6      5.16    
## 17 Rodentia           12.5     NA       
## 18 Scandentia          8.9      0.0025  
## 19 Soricomorpha       11.1      0.000592
```
]
]
---

# Loops are fine

.tr[
Jenny Bryan  
[purrr tutorial](https://jennybc.github.io/purrr-tutorial/index.html)  
[Data Rectangling](https://speakerdeck.com/jennybc/data-rectangling)  
[purrr workshop](https://speakerdeck.com/jennybc/purrr-workshop)
]

---

## For loops

```r
for (value in sequence) {
   statement(s)
}
```

Execute a set of _statements_, once for each item in a vector, array, list, etc. and stop when the last element is reached. 
]

```r
for (x in 1:7) {
  print(x)
} 
```

```
## [1] 1
## [1] 2
## [1] 3
## [1] 4
## [1] 5
## [1] 6
## [1] 7
```
]

---

## Loops and working with groups

```r
msleep %>% 
  group_by(order) %>%   # for order in orders
  slice_sample(1)       # statement to execute 
                        ## (sample element at random)
```

---

# Vectorization is our friend

Operations occur in parallel on all elements, rather than on individual components in sequence

```r
x <- c(1,3,12,4,7,8)
x > 5
```

---

### Logical operators are vectorized (many other fns too!)

```r
x <- c(1,3,12,4,7,8)
x > 5
```

```
## [1] FALSE FALSE  TRUE FALSE  TRUE  TRUE
```

We don't need to type all of this out or a loop

```r
x[1]>5 ; x[2]>5
x[3]>5 ; x[4]>5
x[5]>5 ; x[6]>5
```

---

.center[
<img src="data:image/png;base64,#imgs/dogscopy.jpg" width="36%" />   
]
.center.f4[American Greetings]
---

# If we really need to iterate
.left-column[
<img src="data:image/png;base64,#imgs/purrr.jpeg" width="90%" style="display: block; margin: auto;" />
]

- map functions to data  
- work efficiently with **lists**
- readable code
]
]

.tr[
Thomas Mock (2018)  
[Functional Programming in R with purrr](https://towardsdatascience.com/functional-programming-in-r-with-purrr-469e597d0229)
]

---

.f1.code.center[
map(.red.b[.x], .green.b[.f], ...)
]

apply .b.green.code[.f]

...  arguments for .green.b.code[.f]
]

---

## Mapping functions to data
.center[`map(.x,.f)`]

- an anonymous function, defined on-the-fly

- a formula: unique to `purrr`,  provides a very concise way to define anonymous functions
]

---

## Functions

<img src="data:image/png;base64,#imgs/r-function-syntax.png" width="80%" style="display: block; margin: auto;" />
https://www.learnbyexample.org/r-functions/

---

.b.blue.f2[Function Definition] - The '`function()`' function

.f2.center[`function( arglist ) expr`]

**expression** `(expr)`: 
an object that represents an action that can be performed by R. .b[Expressions contain one or more statements]
]

---

## Defining a function

```r
my_custom_function <- function(my_arg){
  x+1 # statement
}

my_vector <- c(3,5,6,7)

my_custom_function(my_arg = my_vector)
```

```
## [1]  2  4 13  5  8  9
```

---

.f3[The body of the function –the statements that are executed when it runs– is grouped with curly braces   (`{}`)]

```r
f <- function(x) {
  x + 4 # statement
}
```

```r
f(6)
```

```
## [1] 10
```

```r
f(x=6)
```

```
## [1] 10
```
]]

.right-column[
Writing the names of arguments in a function call is optional, but for functions with >1 arguments the order of the argument values matters]

---

```r
f <- function(x) x+4   
f(x=6)
```

```
## [1] 10
```

```r
f <- function(x) mean(x)/2   
f(c(6,43,12,5,7))
```

```
## [1] 7.3
```

---

```r
f <- function(x)  y <- (x*0.2885) log(y) # WON'T RUN  
```

```r
f <- function(x) {
  y <- (x*0.2885) # this is a statement
  log(y)          # this is another statement
}

f(5)
```

```
## [1] 0.3663777
```

---

## Anonymous functions

### Anonymous function syntax

```r
(function(x) x * x + 3)(10)
```

```
## [1] 103
```

the function definition in anonymous functions needs to be wrapped in parentheses to separate function and arguments

---

```r
(function(x) x + 3)(10)
```

Behaves exactly the same as

```r
f <- function(x) x + 3
f(10)
```

```
## [1] 13
```

```r
# or 
f <- function(x) {x + 3}
f(10)
```

```
## [1] 13
```

---

## Anonymous functions

- Also known as _lambda expressions_

- Self-contained

- Comparable with similar approaches in Python
]

---

## Why write our own functions

- Brevity

- Less room for errors

- Take advantage of tools that help us apply functions to many things at once
]

.tr[
Jenny Bryan - Stat545  
[Write your own R functions](https://stat545.com/functions-part1.html)
]

---

# Lists

.f3[Objects which contain elements of different types, such as numbers, strings, vectors, data frames, or even other lists]

```r
my_list <- list(
  a_number = 7,
  a_vector = c("mouse", "cat", "dog"),
  a_dataframe = data.frame(
    a = 1:3, b = c("K", "L", "M")))
```

.tr[
Rebecca Barter (2019)  
[Learn to purrr](https://www.rebeccabarter.com/blog/2019-08-19_purrr/)
]

---

```r
my_list
```

```
## $a_number
## [1] 7
## 
## $a_vector
## [1] "mouse" "cat"   "dog"  
## 
## $a_dataframe
##   a b
## 1 1 K
## 2 2 L
## 3 3 M
```

---

## List indexing
.center[
<img src="data:image/png;base64,#imgs/listspepper.png" width="76%" />   
]
Hadley Wickam (2015) on [Twitter](https://twitter.com/hadleywickham/status/643381054758363136?s=20&t=0Gu4Qs757o5KHEJR2ZwkEg)

---

Group and split with `dplyr` into a list of data frames, one for each group

```r
cars_country <- 
  gt::gtcars %>% group_by(ctry_origin) %>%
  group_split()
```

```r
class(cars_country)
```

```
## [1] "vctrs_list_of" "vctrs_vctr"    "list"
```

---

```r
gt::gtcars %>% group_by(ctry_origin) %>%
  group_keys()
```

```
## # A tibble: 5 × 1
##   ctry_origin   
##   <chr>         
## 1 Germany       
## 2 Italy         
## 3 Japan         
## 4 United Kingdom
## 5 United States
```

---

```r
cars_country[[1]] %>% 
  select(model,ctry_origin) %>% head
```

```
## # A tibble: 6 × 2
##   model    ctry_origin
##   <chr>    <chr>      
## 1 6-Series Germany    
## 2 i8       Germany    
## 3 M4       Germany    
## 4 M5       Germany    
## 5 M6       Germany    
## 6 R8       Germany
```

---

```r
cars_country[[2]] %>% 
  select(model,ctry_origin, hp, msrp) %>% 
  head
```

```
## # A tibble: 6 × 4
##   model        ctry_origin    hp   msrp
##   <chr>        <chr>       <dbl>  <dbl>
## 1 458 Speciale Italy         597 291744
## 2 458 Spider   Italy         562 263553
## 3 458 Italia   Italy         562 233509
## 4 488 GTB      Italy         661 245400
## 5 California   Italy         553 198973
## 6 GTC4Lusso    Italy         680 298000
```

---

.center[
<img src="data:image/png;base64,#imgs/purrrPost.png" />   
]
Luis (2017)
Export Iteratively  
https://luisdva.github.io/rstats/export-iteratively/

---

# Why lists?

- Flexible, can store many kinds of objects

- We can easily create lists by splitting existing objects

- List elements can be named

---

## Mapping functions to data

- Apply functions to many things at once

- Reading many files at once

- Modifying and exporting multiple objects

---

### Apply functions to many things at once

```r
library(openintro)
library(purrr)
data("starbucks")
# create three vectors (numeric)
fat <- starbucks$fat
carb <- starbucks$carb
protein <- starbucks$protein
# put them in a list
nutrients <- list(fat,carb,protein)
# apply the mean function to each list element
map(nutrients,mean)
```

---

### Apply (custom) functions to many things at once

```r
# fn that gets min, max, and range

getMinMaxRange <- function(x){
  min_x <- min(x)
  max_x <- max(x)
  maxmin <- max_x - min_x
  c(min_x, max_x, maxmin)
}
```

---

### Check that the function works for vectors

```r
getMinMaxRange(protein)
```

```
## [1]  0 34 34
```

```r
getMinMaxRange(carb)
```

```
## [1] 16 80 64
```

```r
getMinMaxRange(fat)
```

```
## [1]  0 28 28
```

---

### Use `map()` to apply the fn to all the list elements

```r
map(.x = nutrients, .f = getMinMaxRange)
```

```
## [[1]]
## [1]  0 28 28
## 
## [[2]]
## [1] 16 80 64
## 
## [[3]]
## [1]  0 34 34
```

---

## Type-specific `map()` 
.f3[
`map()` always returns lists

`map_lgl()` returns a logical

`map_int()` returns an integer vector

`map_dbl()` returns a double vector

`map_chr()` returns a character vector

`map_df()` returns a data frame
]

---

```r
map(nutrients, median)
```

```
## [[1]]
## [1] 13
## 
## [[2]]
## [1] 45
## 
## [[3]]
## [1] 7
```

```r
map_dbl(nutrients, median)
```

```
## [1] 13 45  7
```
---

## What about anonymous functions and formulas?

.f3[Once we know how to define new functions, we can write anonymous functions to express what we want to do if we only need to do it once.]

.tr[Anonymous Functions in R - Part 1  
https://coolbutuseless.github.io/2019/03/13/anonymous-functions-in-r-part-1/]

---

## Anonymous functions and formulas

Same behavior

```r
numlist <- list(c(5:10),c(20:27))
# anonymous function
map(numlist, function(x) x+2)
# formula notation
map(numlist, ~ .x + 2)
```

---

.f2[In formula shorthand notation (defined with `~`) 
.b.code.red[.x] always refers to the element of the vector or list we are iterating on]

---

# Reading or writing many files at once

.center.f2[`map(.x,.f)`]

.f2.red.b[`.x`] .f2[can be a list or vector of file paths]

.f2.green.b[`.f`] .f2[can be a function that reads or writes files]

---

### Let's create a directory full of text files

.f3.white[
- load `openintro` and make the `nba_players_19` data available

- group the data by `team` then use `group_split()` to create a list of data frames, one for each team

- create a vector of team names
]

> 😲 We can iterate over two inputs simultaneously

---

# Mapping w/ two inputs

.center.f2[`map2(.x, .y., .f)`]

.pull-left[
<img align="middle" src="data:image/png;base64,#imgs/map2.png" width="87%" />    
]
.pull-right[
Altman et al. (2021)  
[Map with multiple inputs](https://dcl-prog.stanford.edu/purrr-parallel.html)
 ]
 
---

.f3.white[
- Create the folder "nbateams" inside your "Data" folder

- Use `here()` to build file (safe and portable) paths for each team in the vector of team names

- Feed the list of data frames and the vector of file paths to `readr::write_csv`
 ]

---
class: inverse
## Reading many files at once

.f3.white[
- use `dir_ls()` to create a vector with the paths for your newly created csv files

- read each element of this vector with `map()` and `readr::read_csv()`

- BONUS CHALLENGE: type-specific mapping (putting everything into a data frame) will throw an error here. Why?
]

---
class: dk-section-title
background-image:url("imgs/peru.jpg")
background-size: cover

# Regular expressions and working with strings

---

# What are Regular Expressions?

- Also abbreviated as regex, R.E., or regexp (singular)

- A concise language for describing patterns of text

## Specially encoded 🧶.helvetica[_strings_] of characters that match patterns in other text strings

---

## Regular expressions

- Input as a text string that compiles into a mini program built specifically to identify a pattern.

- Can be used to match, search, replace, or split text 
]
---

## 🧶Strings

```r
test_string <- "This sentence is a string."
```

```r
pets <-  c("dogs","cat","parrot","pig")
```

---

## Strings

.pull-right[
<div id="gaxecqetfl" style="overflow-x:auto;overflow-y:auto;width:auto;height:auto;">
<style>html {
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

#gaxecqetfl .gt_table {
  display: table;
  border-collapse: collapse;
  margin-left: auto;
  margin-right: auto;
  color: #333333;
  font-size: 16px;
  font-weight: normal;
  font-style: normal;
  background-color: #FFFFFF;
  width: auto;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #A8A8A8;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #A8A8A8;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
}

#gaxecqetfl .gt_heading {
  background-color: #FFFFFF;
  text-align: center;
  border-bottom-color: #FFFFFF;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#gaxecqetfl .gt_title {
  color: #333333;
  font-size: 125%;
  font-weight: initial;
  padding-top: 4px;
  padding-bottom: 4px;
  border-bottom-color: #FFFFFF;
  border-bottom-width: 0;
}

#gaxecqetfl .gt_subtitle {
  color: #333333;
  font-size: 85%;
  font-weight: initial;
  padding-top: 0;
  padding-bottom: 6px;
  border-top-color: #FFFFFF;
  border-top-width: 0;
}

#gaxecqetfl .gt_bottom_border {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#gaxecqetfl .gt_col_headings {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
}

#gaxecqetfl .gt_col_heading {
  color: #333333;
  background-color: #FFFF00;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 6px;
  padding-left: 5px;
  padding-right: 5px;
  overflow-x: hidden;
}

#gaxecqetfl .gt_column_spanner_outer {
  color: #333333;
  background-color: #FFFF00;
  font-size: 100%;
  font-weight: normal;
  text-transform: inherit;
  padding-top: 0;
  padding-bottom: 0;
  padding-left: 4px;
  padding-right: 4px;
}

#gaxecqetfl .gt_column_spanner_outer:first-child {
  padding-left: 0;
}

#gaxecqetfl .gt_column_spanner_outer:last-child {
  padding-right: 0;
}

#gaxecqetfl .gt_column_spanner {
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: bottom;
  padding-top: 5px;
  padding-bottom: 5px;
  overflow-x: hidden;
  display: inline-block;
  width: 100%;
}

#gaxecqetfl .gt_group_heading {
  padding: 8px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
}

#gaxecqetfl .gt_empty_group_heading {
  padding: 0.5px;
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  vertical-align: middle;
}

#gaxecqetfl .gt_from_md > :first-child {
  margin-top: 0;
}

#gaxecqetfl .gt_from_md > :last-child {
  margin-bottom: 0;
}

#gaxecqetfl .gt_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  margin: 10px;
  border-top-style: solid;
  border-top-width: 1px;
  border-top-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 1px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 1px;
  border-right-color: #D3D3D3;
  vertical-align: middle;
  overflow-x: hidden;
}

#gaxecqetfl .gt_stub {
  color: #333333;
  background-color: #FFFFFF;
  font-size: 100%;
  font-weight: initial;
  text-transform: inherit;
  border-right-style: solid;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
  padding-left: 12px;
}

#gaxecqetfl .gt_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#gaxecqetfl .gt_first_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
}

#gaxecqetfl .gt_grand_summary_row {
  color: #333333;
  background-color: #FFFFFF;
  text-transform: inherit;
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
}

#gaxecqetfl .gt_first_grand_summary_row {
  padding-top: 8px;
  padding-bottom: 8px;
  padding-left: 5px;
  padding-right: 5px;
  border-top-style: double;
  border-top-width: 6px;
  border-top-color: #D3D3D3;
}

#gaxecqetfl .gt_striped {
  background-color: rgba(128, 128, 128, 0.05);
}

#gaxecqetfl .gt_table_body {
  border-top-style: solid;
  border-top-width: 2px;
  border-top-color: #D3D3D3;
  border-bottom-style: solid;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
}

#gaxecqetfl .gt_footnotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#gaxecqetfl .gt_footnote {
  margin: 0px;
  font-size: 90%;
  padding: 4px;
}

#gaxecqetfl .gt_sourcenotes {
  color: #333333;
  background-color: #FFFFFF;
  border-bottom-style: none;
  border-bottom-width: 2px;
  border-bottom-color: #D3D3D3;
  border-left-style: none;
  border-left-width: 2px;
  border-left-color: #D3D3D3;
  border-right-style: none;
  border-right-width: 2px;
  border-right-color: #D3D3D3;
}

#gaxecqetfl .gt_sourcenote {
  font-size: 90%;
  padding: 4px;
}

#gaxecqetfl .gt_left {
  text-align: left;
}

#gaxecqetfl .gt_center {
  text-align: center;
}

#gaxecqetfl .gt_right {
  text-align: right;
  font-variant-numeric: tabular-nums;
}

#gaxecqetfl .gt_font_normal {
  font-weight: normal;
}

#gaxecqetfl .gt_font_bold {
  font-weight: bold;
}

#gaxecqetfl .gt_font_italic {
  font-style: italic;
}

#gaxecqetfl .gt_super {
  font-size: 65%;
}

#gaxecqetfl .gt_footnote_marks {
  font-style: italic;
  font-weight: normal;
  font-size: 65%;
}
</style>
<table class="gt_table">
  
  <thead class="gt_col_headings">
    <tr>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1">drink</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1">price</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td class="gt_row gt_left" style="background-color: #FFFF00;">Coffee</td>
<td class="gt_row gt_right">3.50</td></tr>
    <tr><td class="gt_row gt_left" style="background-color: #FFFF00;">Tea</td>
<td class="gt_row gt_right">2.99</td></tr>
    <tr><td class="gt_row gt_left" style="background-color: #FFFF00;">Juice</td>
<td class="gt_row gt_right">3.20</td></tr>
  </tbody>
  
  
</table>
</div>
]

<br/>
> Remember how we matched patterns in column names when working with `select()` and the `matches()` selection helper

---

## Strings in R are _case sensitive_

Uppercase and lowercase letters are treated as distinct

```r
"rat"=="rat"
```

```
## [1] TRUE
```

```r
"rat"=="rAT"
```

```
## [1] FALSE
```

---

### Regex searches are case sensitive by default

Our options:

- Turn off case sensitivity in the matching,

- Modify the input text before matching. 
]

---
## Why Learn Regular Expressions?

- Save  time when we need to match, extract, or transform text by matching words, characters, and patterns

- Regular expressions can often **replace dozens of lines of code**

- Not specific to any particular programming language or command-line tool
]

---

## Getting started

.fl.w-third[
<div class="f2 mt1 pt0 code">Regexp</div>
</br>
]

.fl.w-third[
<div class="f2 mt0 pt0 code">Input string</div>
</br>
]

.fl.w-third[
<div class="f2 mt0 pt0 code">Match</div>
</br>
]
</br>

</br></br>

The matching is done by the regex engine. We do not access the engine directly, but functions that take regular expressions as arguments call on it whenever its needed.

---

## Getting started

.fl.w-third[
<div class="f2 mt1 pt0 code">Regexp</div>
</br>
`dog`
]

.fl.w-third[
<div class="f2 mt0 pt0 code">Input string</div>
</br>
"The dog is fat."
]

.fl.w-third[
<div class="f2 mt0 pt0 code">Match</div>
</br>
The .bg-yellow[dog] is fat.

]
</br></br></br></br></br>
.f3[
To search for a specific sequence of characters, the regexp (regular expression) we need is simply that sequence of characters]

> find a **d** followed by an **o** followed by a g. (all characters together and in that order when read from left to right)

---

## Online regex testers
.f3[Is the regular expression matching anything?]

- [rubular](https://rubular.com/) by Michael Lovitt

- [regex101](https://regex101.com/) by Firas Dib

- [regexr](https://regexr.com/) by Grant Skinner

- [regexpal](https://www.regexpal.com/) - part of Dan's Developer Tools

---

## Regex Testers

---

.f3.white[Navigate to [rubular](https://rubular.com/), [regex101](https://regex101.com/), [regexr](https://regexr.com/), or [regexpal](https://www.regexpal.com/)

Input the following text into the test string field: 
]

milk  
rice   
cereal  
salt  
soda

.f3.white[Write regular expressions (one at a time) to match the things we can drink and look at the match results]

---

## Literal Characters and Metacharacters

For regular expressions, characters can be grouped into two classes depending on their behavior
]

---

## Literal characters

We will get a match whenever the characters .orange[`d`], .orange[`o`], and .orange[`g`] occur consecutively in the input text]

`"dog"` tells the regex engine:
find a .orange[d], immediately followed by an .orange[o], immediately followed by a .orange[g] (.b[together and in that particular order])

---

## Literal characters (continued)

.f2[They stand for exactly what they are: .orange[`d`] in the regex matches a “.purple.b[d]” in the input text, .orange[`o`]  matches an “.purple.b[o]” in text, and so on.]

---
class: middle

.f2[The power and flexibility of regular expressions comes from their ability to describe more complex patterns.

If a text pattern can be described verbally, we can most likely write regular expression to match it.]

---

### Possible patterns

- "dog" but not "dogs"

- "dogs" but only if the match is at the start of the string,

- digits (0, 1, 2, 3, 4, 5, 6, 7, 8,  9)

- "modeling" or "modelling"  (alternate spellings of the same word)

- words ending in "at" (words that begin or end with a specific pattern)

- strings that start with digits

- dates

---

### Possible patterns

- zip codes

- numbers inside square brackets and also the brackets

- valid Twitter handles (start with @, no spaces or symbols, <16 characters)

- UPPERCASE words

---

## Metacharacters

.f3[To match more than literal strings of text, a small subset of characters that have special functionalities when they appear in a regular expression. Metacharacters do not stand for themselves, they interpreted in a special way.]

.f3[ 
Metacharacters include: .b.orange[[]\^$.|?*+()], which are reserved for unique matching purposes.
]

---

&nbsp; .f1.orange.pre[&nbsp;.&nbsp; ] &nbsp;&nbsp; match any character once
***

.b.orange[f..l] matches ".b.bg-yellow[fill]" and ".b.bg-yellow[fool]", but not "flail"

.b.orange[.top] matches ".b.bg-yellow[stop]" and "is.b.bg-yellow[otop]e", but not "topple"

> note the substring match in "is**otop**e"

---

&nbsp;.b.orange.pre[&nbsp;[ ] &nbsp;] &nbsp;&nbsp; match a set of characters  
***

.b.orange[[cb]at] matches ".b.bg-yellow[cat]", and ".b.bg-yellow[bat]", but not "rat"

.b.orange[[CK]at[iey]] matches ".b.bg-yellow[Caty]", ".b.bg-yellow[Kati]", ".b.bg-yellow[Katy]", and ".b.bg-yellow[Kate]"

---

.f3[
.b[Negation tokens]&nbsp;  
  
.orange.b[[^]] 	&nbsp;&nbsp; match characters not in the specified character set]

> `^` must be the first character inside the brackets

***

.b.orange[[^aoeiou]] matches consonants

.b.orange[[^R]] matches everything except capital R

---

Dash (.b.orange[\-]) inside a character set abbreviates alphabetical or numeric sequences 
***

.b.orange[[A-D]]  matches any single letter from A,B,C, or D (uppercase)

.b.orange[[5-8]]  &nbsp;matches any single digit between 5 and 8

.b.orange[[A-Za-z]]  matches all alphabetical characters

> Character sets are case sensitive

> character ranges can also be negated with `^`

---

&nbsp;.orange.f3.pre[&nbsp;^&nbsp;] &nbsp;&nbsp;     starts with  
&nbsp;.f3.orange.pre[&nbsp;$&nbsp;]  &nbsp;&nbsp; ends with  
.b.orange[`^`] outside a pair of square brackets is an anchor
***

.b.orange[^mil] matches ".b.bg-yellow[mil]kshake" but not "family"

.b.orange[ing$] matches "go.b.bg-yellow[ing]" but not "ingest"

> note the substring matches

---

### Practice
.f3.white[
- Write a regexp that can match **tail**, **tool**, **tall**, and **toil**

- How can we match **Jocelyn**, **Jocelin**, and **Joselyn** but not Jozelyn using character sets?

.fl.w-third.pa0[.tc[
a) Joscel[iy]n
]]
.fl.w-third.pa0[.tc[
b) Jo[sc^z]el[yi]n
]]
.fl.w-third.pa0[.tc[
c) Jo[sc]el[yi]n
]]
]

---

.f3.white[
- Which of these regular expressions matches **`food`** at the beginning of a string?

.fl.w-25[.tl[
a) ^food
]]
.fl.w-25[.tl[
b) food
]]
.fl.w-25[.tl[
c) $food
]]
.fl.w-25[.tl[
d) food^
]]
]

---

.f3[**Quantifiers** specify how many times a character or character class must appear in the input for a match to be found]

&nbsp;.f2[.orange.f1[&nbsp;?&nbsp;] &nbsp;&nbsp;Zero or one ]  	
&nbsp;.f2[.orange.f1[&nbsp;\\\*&nbsp;] &nbsp;&nbsp;Zero or more occurrences] 	
&nbsp;.f2[.orange.f1[&nbsp;+&nbsp;]	 &nbsp;&nbsp;One or more occurrences]   
&nbsp;.f2[.orange.f1[&nbsp;{}] 	&nbsp;&nbsp;Exactly the specified number of occurrences]

> quantifiers apply to the preceding character

---

### Quantifiers

.b.orange[modell?ing] matches ".b.bg-yellow[modeling]" and ".b.bg-yellow[modelling]"

> zero or one `els (l)`

.b.orange[ya*y!] matches ".b.bg-yellow[yy!]", ".b.bg-yellow[yay!]", ".b.bg-yellow[yaaay!]", ".b.bg-yellow[yaaaaaay!]", etc.

> zero or more `aes (a)`

---

### Quantifiers

.b.orange[no+] matches “.bg-yellow.b[no]”, “.bg-yellow.b[nooo]", ".bg-yellow.b[noooooo]", etc, but not “n”

> one or more `oes (o)`

.b.orange[e{2}] matches .“k.b.purple.bg-yellow[ee]p” and “b.b.purple.bg-yellow[ee]" but not “meat”

> exactly two `ees (e)`

---
class: inverse
## Practice

.f3.white[
- Use a quantifier to match **cute**, **cuuute**, **cuuuuuute**, and **cuuuuuuuuute**

- How can we match Computer, computer, Computers, and computers?

.fl.w-third.pa0[.tc[
a) [cC]omputers?
]]
.fl.w-third.pa0[.tc[
b) Computers+
]]
.fl.w-third.pa0[.tc[
c) [cC]omputer[s]+
]]

]
---

.f3[
.b[Alternation] tokens separate a series of alternatives]  
.f3[.orange.b.pre[&nbsp;|&nbsp;] &nbsp;&nbsp; either or
]
***

.b.orange[dog|bird] matches ".b.purple.bg-yellow[dog]" or ".b.purple.bg-yellow[bird]"

.b.orange[gr(a|e)y] matches ".b.purple.bg-yellow[gray]" and ".b.purple.bg-yellow[grey]"
> note the alternation enclosed in brackets

---

.f3[
.b[Special sequences and escapes]<br>
.b.orange.pre[&nbsp;\\&nbsp;] 	&nbsp;&nbsp; signals a shorthand sequence or gives special characters a literal meaning
]

---

### Escapes
.b.orange[hello\\\\\\\\?] matches ".b.purple.bg-yellow[hello?]"

_Metacharacters inside a character set are stripped of their special nature_

---

.b.orange[\w] &nbsp;letters, underscore, and numbers  <br>
.b.orange[\d] &nbsp;digits<br>
.orange.b[\t]  &nbsp;&nbsp; tab &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; .orange.b[\n]  &nbsp; new line &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; .orange.b[\s]  &nbsp;space<br>
.orange.b[\b]  &nbsp;word boundary<br>

>.small[Predefined character classes help us avoid malformed character sets]

---

### Word boundaries .b.orange[`\\b`]

Match **positions** between a _word_ character (letter, digit or underscore) and a non-word character (usually a space or the start/end of a string).
***

> Before a sequence of word characters

.b.orange[\\bcase] matches ".b.bg-yellow.purple[case]" and "two .b.bg-yellow.purple[case]s" but not "suitcase"

> After a sequence of word characters

.b.orange[org\\b] matches "cyb.b.bg-yellow.purple[org] rebellion" but not "organic"

---

### _Word_ characters .b.orange[`\\w`]

Matches any character (letter, digit or underscore). Useful in combination with word boundaries and quantifiers.
***

Equivalent to .orange[\[a-zA-Z0-9_\]]

.b.orange[\\w] matches letters (case insensitive), numbers, and underscores in:

.b.bg-yellow.purple[F33d].%.b.bg-yellow.purple[the_mÖusE] .b.bg-yellow.purple[pl]@.b.bg-yellow.purple[ase]!!#

---

.f3.white[
Enter "That athlete is at the squat rack" as the test string in a regex tester.

Explain the matches obtained with the following three regular expressions?
 
 a) at  
 b) \bat  
 c) at\b
]
---

## Anchor, wildcard with quantifier (0 or more)

.b.orange[^can.\\\*] matches ".b.purple.bg-yellow[can]", ".b.purple.bg-yellow[can]ine", ".b.purple.bg-yellow[can]oli", and ".b.purple.bg-yellow[can]adian"

##  Wildcard and quantifier
.b.orange[A.\\*x] strings that start with "**A**" and end with "**x**"

---

## Shorthand sequence (space) and quantifier

.b.orange[\\s{3}] matches three spaces

## Anchors, character set, and quantifier (one or more)

.b.orange[^[a-z]+$] matches a lowercase string

---

## Word characters, quantifiers, word boundaries, and anchors

.b.orange[\w+\b$] matches the last word in a string

"Fix the .b.purple.bg-yellow[car]."  
"12 .b.purple.bg-yellow[eggs]"

.b.orange[^\w+\b] matches the first word in a string

".b.purple.bg-yellow[Fix] the car."  
".b.purple.bg-yellow[12] eggs"

---

<img src="data:image/png;base64,#imgs/regular-expressions.svg" width="66%" style="display: block; margin: auto;" />
</br>
.center.small[Modified from '[Regular Expressions](https://github.com/rstudio/concept-maps/#regular-expressions)' concept map by Greg Wilson]

---
class: inverse
## Practice

.f3.white[
Write regular expressions to match the following, making sure they work in an interactive regex tester:

1. strings with both letters and digits
2. strings ending in a period
3. lowercase strings
4. exactly four digits
5. strings without the letter _**s**_
]

---
class: center, middle, dk-section-title
background-image:url("imgs/bsas.jpg")
background-size: cover
# Regex in R

---

---

# 📦 .red[`stringr`]

- Function names start with .b[`str_`]

- All functions take a vector of strings as the first argument (pipe-friendly)

**`regex()`** modifier to control matching behavior

---

## `stringr` examples

**Matches?**

```r
str_detect(string = c("catalog", "battlecat", "apple"), pattern = "cat")
```

```
## [1]  TRUE  TRUE FALSE
```

Output is a logical, TRUE or FALSE vector of the same length as our input string

---

## `stringr` examples

**Which elements contain matches?**

```r
str_which(string = c("catalog", "battlecat", "apple"), pattern = "cat")
```

```
## [1] 1 2
```

Output is the index for each of the matching elements

---

## `stringr` examples

**Replacing matches**

```r
str_replace(string = c("colour", "neighbour", "honour"),
            pattern = "ou",
            replacement = "o")
```

```
## [1] "color"    "neighbor" "honor"
```

---

## `stringr` examples

**Case insensitive matching**

```r
str_replace(string = c("colOur", "neighboUr", "honOUr"),
            pattern = regex("ou", ignore_case = TRUE),   
            replacement = "o") 
```

---

## `stringr` examples

**Case insensitive matching**

```r
str_replace(string = c("colOur", "neighboUr", "honOUr"),
*           pattern = regex("ou", ignore_case = TRUE),
            replacement = "o") 
```

---

## `stringr` examples

**Case insensitive matching**

```r
str_replace(string = c("colOur", "neighboUr", "honOUr"),
*           pattern = regex("ou", ignore_case = TRUE),
            replacement = "o") 
```

```
## [1] "color"    "neighbor" "honor"
```

---

## Practice

Regular expressions
1. ^dog
2. ^[a-z]+$
3. \\\\d
]

```r
test_vector <- c("Those dogs are small.","dogs and cats",
                 "34","(34)","rat","watchdog","placemat",
                 "BABY","2011_April","mice")
```

---

## Using regular expressions in data manipulation

---

## Cleaning data with regex

To select variables in 📦 .b.red[`dplyr`] and 📦 .b.red[`tidyr`], we:

- write out their names

- refer to them by position

- specify ranges of contiguous variables

- .f3[use 📦 .b.red[`tidyselect`] helper functions]

---

## 📦 .red.b[`tidyselect`] helpers

**`starts_with()`**: Starts with a prefix

**`ends_with()`**: Ends with a suffix

**`contains()`**: Contains a literal string

---

## 📦 .red.b[`tidyselect`] helpers

---

### Selecting columns by name

`penguins` data from 📦 .red.b[`palmerpenguins`]

```r
names(penguins)
```

```
## [1] "species"           "island"            "bill_length_mm"   
## [4] "bill_depth_mm"     "flipper_length_mm" "body_mass_g"      
## [7] "sex"               "year"
```

```r
penguins %>% 
  select(species, bill_length_mm, flipper_length_mm) %>% sample_n(3)
```

```
## # A tibble: 3 × 3
##   species bill_length_mm flipper_length_mm
##   <fct>            <dbl>             <int>
## 1 Gentoo            46.2               221
## 2 Gentoo            44.5               217
## 3 Gentoo            45.3               208
```
]

---

### Selecting columns by matches in variable names

```r
penguins %>% 
  select(species, matches("length")) %>% 
  sample_n(3)
```

```
## # A tibble: 3 × 3
##   species bill_length_mm flipper_length_mm
##   <fct>            <dbl>             <int>
## 1 Gentoo            45.2               223
## 2 Adelie            38.7               195
## 3 Adelie            39.7               193
```

---

### Match values and filter rows

Mammals sleep dataset (`msleep`) from 📦 .red.b[`ggplot2`]

```r
library(ggplot2)
```

```r
msleep %>% select(name,genus) %>% sample_n(4)
```

```
## # A tibble: 4 × 2
##   name                 genus      
##   <chr>                <chr>      
## 1 Round-tailed muskrat Neofiber   
## 2 Chinchilla           Chinchilla 
## 3 Rabbit               Oryctolagus
## 4 Vesper mouse         Calomys
```

---

### Match values and filter rows

Filter to keep rats only

```r
msleep %>% 
  select(name,genus) %>% 
  filter(str_detect(string = name,pattern = "rat"))
```

```
## # A tibble: 5 × 2
##   name                      genus     
##   <chr>                     <chr>     
## 1 African giant pouched rat Cricetomys
## 2 Round-tailed muskrat      Neofiber  
## 3 Laboratory rat            Rattus    
## 4 Cotton rat                Sigmodon  
## 5 Mole rat                  Spalax
```

---

## Practice

```r
msleep %>% 
  select(name,genus) %>% 
  filter(str_detect(string = name,pattern = "rat"))
```